Optical Non-Line-of-Sight Physics-based 3D Human Pose Estimation

We describe a method for 3D human pose estimation from transient images (i.e., a 3D spatio-temporal histogram of photons) acquired by an optical non-line-of-sight (NLOS) imaging system. Our method can perceive 3D human pose by `looking around corners' through the use of light indirectly reflected by the environment. We bring together a diverse set of technologies from NLOS imaging, human pose estimation and deep reinforcement learning to construct an end-to-end data processing pipeline that converts a raw stream of photon measurements into a full 3D human pose sequence estimate. Our contributions are the design of data representation process which includes (1) a learnable inverse point spread function (PSF) to convert raw transient images into a deep feature vector; (2) a neural humanoid control policy conditioned on the transient image feature and learned from interactions with a physics simulator; and (3) a data synthesis and augmentation strategy based on depth data that can be transferred to a real-world NLOS imaging system. Our preliminary experiments suggest that our method is able to generalize to real-world NLOS measurement to estimate physically-valid 3D human poses.Comment: CVPR 2020. Video: https://youtu.be/4HFulrdmLE8. Project page: https://marikoisogawa.github.io/project/nlos_pos

BODY SHAPE AND CENTER OF MASS ESTIMATION USING MULTI-VIEW IMAGES

This study presents a method for estimating human 3D body shape in action. We propose a method for estimating 3D human body shape motion that uses multiple view images and visual hulls. Related methods necessitated lengthier preparations, such as camera calibration, which would require several tries before actually capturing the image. We solve this issue by combining state-of-the-art computer vision methods to automatically process the required inputs and parameters, so that camera images are the only resource needed for estimation. In our experiments, we applied our method to a video of human subject kicking a soccer ball to left and right side of a goal; we successfully acquired the subject’s 3D body shape. In addition, we verified that the application’s automatically obtained body shape successfully provides the subject’s center of mass

Importance Rank-Learning of Objects in Urban Scenes for Assisting Visually Impaired People

This paper examines an importance rank learning method of objects in urban scenes for assisting visually impaired people. Object detection methods have been used to assist visually impaired people in identifying obstacles in urban scenes, such as cars and trees. However, these existing methods are not dedicated to predicting which obstacle is important. Thus, we propose a method that estimates the importance of objects and warns them to users in order of importance ranking. We introduce a neural network-based ranking estimation method to predict the importance ranking of objects. In particular, our method uses optical flow from the previous frame and region data of detected objects as input. It helps to consider states of moving objects (e.g., cars, motorbikes, people) in a scene. Experimental results show that our model outperforms three other baselines qualitatively and quantitatively. Furthermore, our method was highly evaluated than the baseline methods by qualified caregivers of the visually impaired people